CytoAtlas

Why independence correction matters: Pooling across tissues or cancer types inflates correlations through confounding. For example, GTEx pooled CytoSig median ρ (0.211) is 40% higher than the independence-corrected by-tissue value (0.151); SecAct shows +30% inflation (0.394 vs 0.304). All results in this report use the corrected values. For a detailed comparison of pooled vs independent levels, including inflation magnitude and finer cell-type stratification, see the Section 4.1 statistical supplement.

The key insight: CytoAtlas is not trying to replace DL-based tools. It provides an orthogonal, complementary signal that a human scientist can directly inspect. When CytoAtlas says "IFNG activity is elevated in CD8+ T cells from RA patients," you can verify this by checking the IFNG signature genes in those cells.

PRIMARY independent level: The summary table above reports results at each dataset’s PRIMARY independent level — the aggregation level where samples are fully independent (each donor counted once). This ensures correlation statistics are not inflated by donor duplication. See the “Primary Level” column for each dataset’s level.

How “N Targets” is determined: A target is included in the validation for a given atlas only if (1) the target’s signature genes overlap sufficiently with the atlas gene expression matrix, and (2) the target gene itself is expressed in enough samples to compute a meaningful Spearman correlation. Targets whose gene is absent or not detected in a dataset are excluded. CytoSig defines 43 cytokines and SecAct defines 1,170 secreted proteins. Inflammation Atlas Main retains only 33 of 43 CytoSig targets and 805 of 1,170 SecAct targets because 10 cytokine genes (BDNF, BMP4, CXCL12, GCSF, IFN1, IL13, IL17A, IL36, IL4, WNT3A) are not sufficiently expressed in these blood/PBMC samples.

Stratified levels (GTEx by_tissue, TCGA primary_by_cancer): Correlations are computed within each tissue/cancer type (ensuring independence), then summarized across groups. N Targets counts unique targets at the “all” aggregate level. Finer per-tissue or per-cancer breakdowns are available in Section 4.3 below.

Why does SecAct appear to underperform CytoSig in Inflammation Atlas Main?

This is a composition effect, not a genuine performance gap, confirmed by two complementary statistical tests:

Total comparison (Mann–Whitney U test): Compares the full ρ distributions of CytoSig (43 cytokine signatures) vs SecAct (~1,170 secreted protein signatures) using independence-corrected values. For GTEx/TCGA, each target’s representative ρ is the median across per-tissue/cancer values (median-of-medians); for other datasets, donor_only/tumor_only ρ is used directly. SecAct achieves a significantly higher median ρ in 5 of 6 datasets (GTEx: p = 4.76 × 10⁻⁴; TCGA: p = 2.85 × 10⁻³; CIMA: p = 3.18 × 10⁻²; scAtlas Normal: p = 1.04 × 10⁻⁴; scAtlas Cancer: p = 1.06 × 10⁻⁵). Inflammation Atlas Main is the sole exception (U = 14,101, p = 0.548, not significant) and the only dataset where CytoSig’s median ρ (0.323) exceeds SecAct’s (0.173).

Matched comparison (Wilcoxon signed-rank test): Restricts to the 32 targets shared between both methods (22 direct + 10 alias-resolved), each target serving as its own control. SecAct’s median ρ is consistently higher across all 6 datasets, reaching significance in 5 (GTEx: p = 3.54 × 10⁻⁵; TCGA: p = 3.24 × 10⁻⁶; CIMA: p = 2.28 × 10⁻²; scAtlas Normal: p = 3.54 × 10⁻⁵; scAtlas Cancer: p = 3.54 × 10⁻⁵). Inflammation Atlas Main is not significant (p = 0.141).

Inflammation Atlas Main is largely blood-derived, so many SecAct targets that perform well in multi-organ contexts contribute near-zero or negative correlations here. In fact, 99 SecAct targets are negative only in Inflammation Atlas Main but positive in all other datasets, reflecting tissue-specific expression limitations rather than inference failure. The “Matched” tab above demonstrates the fair comparison on equal footing.

Stratified validation: Instead of aggregating tissues/cancers into a single median-of-medians, this view shows the CytoSig vs SecAct comparison within each individual tissue (GTEx) or cancer type (TCGA). Mann-Whitney U test (Total tab: all targets) and Wilcoxon signed-rank test (Matched tab: 32 shared targets) with BH-FDR correction across all strata within each dataset.

Key insight: On the 32 matched targets, SecAct wins direction in every stratum — 29/29 GTEx tissues and 33/33 TCGA cancer types — with 25/29 and 31/33 reaching significance (q<0.05). This unanimous result across 62 independent strata rules out Simpson’s paradox. On total targets, SecAct wins in 28/29 GTEx tissues (21 significant) and 30/33 TCGA cancers (15 significant); the few CytoSig-favored strata (Brain in GTEx; Kidney Chromophobe, Ovarian, Uveal Melanoma in TCGA) are all non-significant. Since SecAct outperforms CytoSig on the same 32 cytokines, the advantage is not about target breadth but about signature quality. SecAct’s spatial-transcriptomics-derived signatures (Visium) capture tissue-context-dependent cytokine regulation that CytoSig’s case-control cytokine treatment experiments might not capture. The advantage is largest in tissues with complex cellular microenvironments (GTEx: Small Intestine Δ=+0.47, Esophagus +0.41; TCGA: Testicular +0.33, Cervical +0.32) and smallest in homogeneous contexts (GTEx: Breast +0.001, Pituitary +0.06; TCGA: Brain Glioma +0.06, Kidney Chromophobe +0.09).

Cross-platform concordance: This section tests whether expression–activity relationships replicate across measurement technologies. For each tissue (GTEx) or cancer type (TCGA), we compute per-target Spearman ρ from bulk RNA-seq data and compare it to the same target’s ρ from single-cell pseudobulk data (scAtlas). Wilcoxon signed-rank tests (paired by target) with BH-FDR correction assess whether ρ values differ between platforms.

Key finding: Using all targets, SecAct shows significant bulk–pseudobulk differences in most strata (11/13 GTEx tissues, 5/11 TCGA cancers), while CytoSig shows almost none (1/13, 0/11). However, the Matched tabs reveal this is a statistical power effect, not a signal quality difference: when restricted to the same 32 shared targets, both CytoSig and SecAct show no significant platform differences (0/13 and 0/13 for GTEx; 0/11 and 1/11 for TCGA). The apparent platform sensitivity in SecAct’s full panel is a statistical power effect, not a signal quality difference: matched and unmatched SecAct targets show the same per-target platform shift (mean |Δ| = 0.298 vs 0.302, Mann–Whitney p = 0.82), but SecAct’s ~1,000 paired targets per tissue provide 25× more observations than CytoSig’s ~40, easily detecting the same tiny systematic shift (Δ ≈ 0.03) that CytoSig lacks power to detect. Core cytokine targets are platform-robust.

Key insight: None of these targets have isoforms or subunits mapping to different gene IDs that would cause gene ID confusion. The poor correlations are supposedly driven by post-translational regulation (membrane shedding, proteolytic activation, decoy receptor sequestration), paracrine signaling topology (producer and responder cells are different cell types), and heteromeric complex dependence (LTA requires LTB). These represent fundamental limitations of correlating ligand mRNA abundance and predicted activity as validation strategy of cytokine activity prediction model.

Aggregation levels explained: Pseudobulk profiles are aggregated at increasingly fine cell-type resolution. At coarser levels, each pseudobulk profile averages more cells, yielding smoother expression estimates but masking cell-type-specific signals. At finer levels, each profile is more cell-type-specific but based on fewer cells.

How each correlation value is computed: For each (target, atlas) cell, we compute Spearman rank correlation between predicted cytokine activity (ridge regression z-score) and target gene expression across all donor-level pseudobulk samples. Specifically:

1. Pseudobulk aggregation: For each atlas, gene expression is aggregated to the donor level (one profile per donor or donor × cell type).
2. Activity inference: Ridge regression (secactpy.ridge, λ=5×10⁵) is applied using the signature matrix (CytoSig: 4,881 genes × 43 cytokines; SecAct: 7,919 genes × 1,170 targets) to predict activity z-scores for each pseudobulk sample.
3. Correlation: Spearman ρ is computed between the predicted activity z-score and the original expression of the target gene across all donor-level samples within that atlas. A positive ρ means higher predicted activity tracks with higher target gene expression.

GTEx uses per-tissue pseudobulk (median-of-medians across 29 tissues); TCGA uses per-cancer type (median-of-medians across 33 cancers); CIMA/Inflammation Atlas Main use donor-only; scAtlas Normal uses donor-only; scAtlas Cancer uses tumor-only.

Ten methods compared on identical matched pairs across 4 combined datasets:

1. CytoSig — 43 cytokines, 4,881 curated genes, global (all cell types)
2. LinCytoSig (orig) — cell-type-matched signatures, all ~20K genes
3. LinCytoSig (gene-filtered) — cell-type-matched signatures, restricted to CytoSig’s 4,881 genes
4. LinCytoSig Best (combined) — best cell-type signature per cytokine (selected by combined GTEx+TCGA bulk ρ), all ~20K genes
5. LinCytoSig Best (comb+filt) — best combined bulk signature, restricted to 4,881 genes
6. LinCytoSig Best (GTEx) — best per cytokine selected by GTEx-only bulk ρ, all ~20K genes
7. LinCytoSig Best (TCGA) — best per cytokine selected by TCGA-only bulk ρ, all ~20K genes
8. LinCytoSig Best (GTEx+filt) — GTEx-selected best, restricted to 4,881 genes
9. LinCytoSig Best (TCGA+filt) — TCGA-selected best, restricted to 4,881 genes
10. SecAct — 1,170 secreted proteins (Moran’s I), subset matching CytoSig targets

Key findings:

• SecAct achieves the highest median ρ across all 4 combined datasets, benefiting from spatial-transcriptomics-derived signatures.
• CytoSig outperforms most LinCytoSig variants at donor level, with one notable exception: scAtlas Normal Best-orig (0.298) exceeds CytoSig (0.216).
• Gene filtering improves LinCytoSig in most datasets (CIMA +102%, Inflammation Atlas Main), confirming noise reduction from restricting the gene space.
• GTEx-selected best performs comparably to combined-selected in most datasets but slightly better in scAtlas Cancer (0.300 vs 0.275). TCGA-selected best generally underperforms other selection strategies, suggesting GTEx’s broader tissue coverage provides more generalizable selections.
• Gene filtering of GTEx/TCGA-selected: GTEx+filt and TCGA+filt show mixed results — filtering sometimes improves (e.g., TCGA+filt in Inflammation Atlas Main: 0.260 vs TCGA-orig 0.168) but can also reduce performance, indicating the optimal gene space depends on both the selection dataset and target dataset context.
• General ranking: SecAct > CytoSig > LinCytoSig Best variants > LinCytoSig (filt) > LinCytoSig (orig), though dataset-specific exceptions exist.

Key finding: Lineage-specific aggregation provides no systematic advantage at any level.

• At every level, LinCytoSig underperforms CytoSig (mean Δρ ranges from −0.08 at coarse L1 to −0.02 at fine L4 in CIMA). Finer cell types reduce the gap slightly but never close it.
• SecAct wins at every level in CIMA and scAtlas. In Inflammation Atlas Main L2, LinCytoSig is nearly tied with SecAct (Δρ = +0.01) but still loses to CytoSig.
• Per cell type: Only 5 of 43 cell types show consistent LinCytoSig advantage vs CytoSig (NK Cell, Basophil, DC, Trophoblast, Arterial Endothelial). No cell type beats SecAct.
• Interpretation: CytoSig’s global signature, derived from median log2FC across all cell types, already captures the dominant transcriptional response. Restricting to a single cell type’s response introduces noise from small sample sizes without gaining meaningful lineage specificity. The hypothesis that finer resolution should favor LinCytoSig is not supported by the data.